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SMART MESSENGER 
CROSS-REFERENCE TO RELATED APPLICATIONS 

The subject matter of this patent application is related to that of 
application Serial No. 09/079,754 for "Apparatus and Method for User 
5 Recognition Employing Behavioral Passwords", filed on May 15, 1998, by D. 

Kanevsky and S. Maes, application Serial No. 09/108,544 for "Audio/Video 
Archive and Method for Automatic Indexing and Searching", filed My 1, 
1998, by D. Kanevsky, M. Padmanabhan, AR Zingher, and S. Maes, and 
application Serial No. 09/437,971 for "Methods and apparatus for semantic 
1 0 unit based automatic indexing and searching in data archive systems", filed 

November 10, 1999, by JC Chen and D. Kanevsky both of which are assigned 
to a common assignee herewith. The disclosures of applications Serial No. 
09/079,754, Serial No. 09/108,544, and Serial No. 09/437,971 are 
incorporated herein by reference. 

15 DESCRIPTION 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention generally relates to improvements to network 
messaging systems and, more particularly, to identification, sequencing, and 
20 presentation of user-to-user messages when using services such as Instant 

Messenger or ICQ. 
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Background Description 

There exist several problems with current internet messaging systems 
such as Instant Messenger and ICQ, a free software chat and messaging 
system. The first problem is that when people write long messages and send 
several messages to the same person over a short period of time, and the 
recipient replies to these messages, then several topics may be discussed within 
one session and this may confuse the sender/recipients because it may become 
unclear as to which question the recipient has just replied. For example, a 
person may send a message with the question, "What are you doing tomorrow 
night?", shortly followed by another question, "What are you doing today?" 
The recipient may reply, "I am going to the movies." This response may 
confuse the sender, as to whether the reply was to the first or second question. 
Although this is a simplified example, it is relatively easier to imagine more 
difficult scenarios when several complicated topics are being discussed in a 
session, the questions and replies may get easily confused. The scenario 
becomes infinitely more confusing when several people are involved in a "chat 
session". 

Another typical problem with America On Line (AOL) is that if a 
person is away from their computer and they receive a message during this 
time, while a family member who knows, their screen name, sits at the 
computer, the family member may begin to answer as though they were the 
intended recipient. The disguised identity scenario may happen as a child 
jokingly pretending to be a parent, or a room-mate or friend pretending to be 
the person whose screen name identity they have assumed. 
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SUMMARY OF THE INVENTION 



It is therefore an object of the present invention to provide a new 
means to monitor and detect new message topics between two or more users 
engaged in an on-line messaging communication with new topic messages 
presented in an orderly fashion whereby replies are coordinated and presented 
by topic and by user. 

It is another object of the present invention to monitor the content of 
the communication and provide a level of security against harassing or 
undesirable topics. 

It is yet another object of the invention to monitor the boimetrics of the 
communications and determine if any party is has changed pattern signifying a 
possible change in identity of the sending party. 

It is yet another objective of the invention to notify involved users in 
the communication of a possible change of a sender's identity and present a 
means to validate the identity of the sending party in question. 

According to the invention, there is provided a collection of software 
modules that resides in a user's personal computer and operates in conjunction 
with a messaging service. 

In order to solve the first problem, our invention implements a smart 
topic agent in the messenger. The smart topic agent uses a time synchronizer 
and semantic means to separate several topics within one conversation, from 
both sides (sender, recipient), and allows the system to discriminate which 
message on the part of the sender corresponds with what message from the 
recipient. The system may give similar themes the same color, or open a 
separate window for each separate topic. In our example, the system would 
have understood that the answer was to the first question since it had not yet 
received an answer. The system would have separated in its own window the 



series, "What are you doing tomorrow?", and the answer, "I am going to the 
movies." If the user felt that it may be confusing, he may create a more 
complicated interface by putting two questions on a similar level with arrows 
to the answer, "going to the movies" followed by a question mark. In this way 
the user sees that there is an ambiguity as to which question the answer refers. 

In order to solve the second messenger problem, how to prevent 
disguised screen name identity, the following methods are used. The system 
identifies the typing style of the user, and if the style of the current sender does 
not resemble that of the original screen name user, the system may warn the 
recipient that the sender may not be the original user. The system may also ask 
the current sender a series of questions to which only the original screen name 
user would know the answers. Similarly, people who converse using messenger 
services on a regular basis may agree to answer a set of particular questions to 
which only they know the answers. In this case, the security system of the 
messenger service knows that the sender/recipients must exchange a series of 
questions, and the system knows that if these questions are not exchanged or 
are improperly answered, then, one of the users is incorrect. 

Alternatively, or in addition, biometrics may be used to identify a 
sender. Specifically, voice recognition systems to identify a speaker may be 
used to provide security to the system. Also, using biometrics, an approximate 
age of a user may be estimated to provide an indication whether the sender 
might not be the original screen name user. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better 
understood from the following detailed description of a preferred embodiment 
of the invention with reference to the drawings, in which: 
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Figure 1 is a block diagram showing the major elements of the system; 

Figure 2 is a flow diagram illustrating the logic of the topic separator; 

Figure 3 is a flow diagram illustrating the logic and presentation of the 
messenger graphic user interface (GUI); 

Figure 4 is a flow diagram illustrating the process flow within a user's 
computer; 

Figure 5 is a flow diagram illustrating the process flow of the 
multimedia processing by the messenger service; 

Figure 6 is a block diagram of the security checker function; 

Figure 7 is a flow diagram illustrating the process flow of the 
questionary function; 

Figure 8 is a functional block diagram of the messenger service; and 

Figure 9 is flow diagram showing the overall operation of the invention. 

DETAILED DESCRIPTION OF A PREFERRED 
EMBODIMENT OF THE INVENTION 

Referring now to the drawings, and more particularly to Figure 1, there 
is shown a schematic representation of the invention. The messenger service 
100 is utilized by user 101 and user 102 to exchange messages (e.g., AOL 
instant messenger, ICQ, Net Meeting, and Peer-to-Peer). Module 106 
represents a network, such as the Internet, through which two or more 
computers 105 may be connected. Each of these computers 105 (belonging to 
users 101 and 102) run the messenger GUI (Graphic User Interface) 
application 104 and the topic separator application 103. The topic separator 
application 103 separates the users' messages into related topics. The 
messenger GUI application 104 gives different modes of presentation for 
messages with different topics. For example, each window may receive the 



same topic, or each different topic may be represented by a different color 
(while the same topic is represented by a single color). A computer 105 runs 
the topic separator, messenger GUI, and security provision applications. 
Although Figure 1 shows only two users, many users may be connected during 
a single chat session. 

Figure 2 illustrates the logic of the topic separator application 103. 
User 1, user 2, and user N send text input 203 that associates with the time 
synchronizer 208 that puts a time stamp on the respective message. The time 
synchronizer 208 is connected to the topic message splitter 207 which 
associates the messages sent by different users. Therefore, according to the 
time stamp, it is possible to know what message of one user would relate to 
the message of another user. The time synchronization of messages will be 
explained in detail below. The topic message splitter 207 uses information from 
the topic synchronizer 208 as well as the topic classifier 206. In this way, the 
messages from different users that are associated by topic and time stamp are 
put into a single group. Each group in module 207 is composed of messages 
unified by topic and time stamp. The topic classifier 206 uses module 205, the 
topic change detector, that determines changes in the topic of conversation. 
There are several ways to identify that a topic has changed, as described in 
U.S. Patent No. 6,104,989 to Dimitri Kanevsky et al. for "Real time Detection 
of Topical Changes and Topic Identification via Likelihood Based Methods". 
The topic change detector 205 uses information from the language model 
(LM) processor 204 that receives information from the text input 203. The 
language model processor 204 utilizes a variety of semantic processors and 
language models that are associated with the text input, semantic 
understanding, and probabilities. These probabilities are used in the topic 
change detector 205. For example, if two messages are sent at different times 
but seem to have a mild topical relationship they are probably not related. 



Figure 3 illustrates the logic of the messenger GUI application 104. The 
messenger GUI application operates to split messages 300 that enter from the 
topic message splitter 207 (Figure 2). The split messages enter the messenger 
windows per topic 301 . For each split message, a new window is formulated. 
For example, windows 302, 303, and 304 display three different topics that 
may occur between several different users. Window 302 receives messages 
from user 3, 3 14, and from user 1,315. Window 303 receives messages 307 
and 308 from users 1 and 2, 315 and 316, respectively. Users 1 and 2 have 
another message window 310 referring to another topic with different 
messages 350, from user 1, and 309 from user 2. This allows users 1 and 2 to 
have separate windows in which they can discuss their separate topics. They 
may also have windows with messages 311 and 312 that correspond to user 1 . 
User 2 has window 313 that may be tied to either window 3 1 1 and window 
312. This occurs when the topic separator could not decide to which topic the 
message belongs, 3 1 1 or 3 12. Module 306 represents a question mark that may 
be displayed so that the users understand that the system is unsure of which 
topic window to place the new message. 

Another method for designating the different topics is represented by 
module 305, a color menu box that controls the messages between users 1 and 
2 that are represented by reference numerals 315 and 3 16 in module 306. In 
this module, different themes are represented by different colors. For example, 
user 1 has several topics that are red in color, while user 2 may have topics that 
are colored red and blue. Users may have the option to choose the color with 
which their topic is displayed, or this function may be automatically controlled 
by the system. 

Figure 4 illustrates the process performed by the computer 105 
(Figure 1) and the applications that are running on the user's computer. Input 
data is composed mostly of textual data 400, but may consist of images 401, 
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and sound 402. In principle, the message may be a voice message that utilizes 
speech recognition to translate voice messages into text and then use the topic 
separator 405. This possibility is not illustrated in Figure 1 but is possible, as 
may be necessary with visually impaired persons. In this case, a visually 
impaired person may just listen to different topics being read. The text data 
may also be accompanied with images 401. In this case, the topic separator 
must be capable of image interpretation. This means that if a person sends a 
text message accompanied with an image, the system utilizes the time 
synchronizer 410 to identify which topics have images that accompanied the 
text. The sound, image, and text (multimedia) combinations are illustrated in 
greater detail in later figures. 

Module 404 represents the security checker that checks for the 
identification of the message sender. The security checker may also be capable 
of monitoring the content of the messages in order to prevent, for example, 
sexual or other undesirable content for a variety of reasons. Module 406 is the 
statistical collector module. The statistical collector may be stored in the 
database 430, which may be local in the user's computer, or may be stored at 
the messenger server. It may be later used to create a language model and for 
topic change detection, and the topic classification for all of the models. For 
the most part, it is used to create a language model characteristic unique to the 
particular user. Typically, this process occurs on the server. The statistical 
collector 406 also collects local data that is used by the topic separator module 
405 on the next level of the message processing. The control device 407 uses 
information from the topic separator module 405 to control the GUI 
application 408, by creating new windows, directs text to the new windows, 
and controls the colors of different topics. The time synchronizer 410 is used 
by the topic separator 405, as was explained with reference to Figure 3. 



Figure 5 illustrates an example of multimedia processing by the 
messenger server with the input of textual data, voice data and image data. In 
this case, image processing by the image recognition module 500 occurs as is 
described in co-pending patent applications Serial No. 09/108,544 and Serial 
No. 09/437,971. Voice message data 402 are processed by the speech 
recognition module 501. Along with the textual data, the multimedia data 
enters the topic identifier 502 which identifies the topic of each message group. 
The topic identifier functions according to the description in module 206 as 
well as aforementioned U.S. Patent No. 6,104,989. Module 503 is responsible 
for topic matching with the help of the time synchronizer 410. Images and 
multimedia data are matched by topic or by time stamp. For example, if the 
users are discussing cars, and an image of a car is sent, the image is made to 
accompany the respective text. Or, if a vacation is being discussed, a photo of 
a vacation scene may accompany the text. In these cases, the themes are 
matched by topic and/or by the time synchronizer 410 to allow for 
images/multimedia sent at approximately the same time as a particular text was 
written, to be grouped together as a topic match in a single window. The 
multimedia placer 504 decides which window to place a piece of multimedia 
data in order to match the topic of a text sequence. 

Figure 6 illustrates the security checker module 404 shown in Figure 4. 
The verification of user identity module 600 uses biometrics 60 1 as described 
in co-pending patent application Serial No. 09/079,754 for "Apparatus and 
Method for User Recognition Employing Behavioral Passwords". The 
biometric may use sensors that can measure the user's identity through voice, 
images, and other methods. Module 602 examines the user's typing patterns 
and compares them with previously stored typing patterns to determine if the 
current user is the real original one. Module 603 asks the user random 
questions that, when properly answered identify that the user is the real original 
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user, as opposed to improperly answered questions that result in an 
identification of an incorrect user. An earlier U.S. Patent No. 5,774,525 to 
Dimitri Kanevsky et. al. for "Method and Apparatus for Utilizing Dynamic 
Questioning to Provide Secure Access Control " discloses a similar user 
identification system to identify the correct user. The major difference between 
the two systems is that in the prior patent, the system checked the correct 
answer, but in this messenger system the other user checks for the correct 
answer. For this reason, the questions asked by the server 600 may be standard 
for all users, but only the recipient may know the correct answer for the sender 
(or question answerer). For example, the questions may be "what year were 
you born?", "where do you study? 7 ', and similar general questions. Module 603 
may also ask questions that are specific to each user that entered their screen 
name, or to both users; therefore, both users must be capable of answering the 
questions. The information for the questions and answers may be kept on a 
server, either local or remote. 

Figure 7 illustrates module 603 (Figure 6), asking random questions, in 
more detail. It is composed of module 700, the database of user questions, 
both specific and general questions. This module 700 is connected to the 
question module 701, which searches the database for the correct questions, 
functioning according to U.S. Patent No. 5,774,525. Modules 702 and 703 
illustrate the interactions between users. User 1 receives a question from the 
question module 701, the question/answer will then be relayed to user 2. 
Similarly, user 2 receives a question, whose answer is automatically sent 
(without the ability to be changed by user 2) to user 1 for verification of 
identity. If the answers are correct (as decided by the users themselves), then 
the conversation may go on. The general idea is that it is the choice of the 
users to accept the correct identity or ignore the incorrect identity. 
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Figure 8 illustrates several of the functions of the messenger service 
100. The question module 800 is used in combination with the user personal 
data module 801 to ask either general or specific questions of the user. In a 
preferred implementation of the invention, the personal data might be 
maintained in a central repository for storage of this information, but the 
invention does not require a central storage of the user's answers. The personal 
data might also be distributed. Questions are asked of the users in an 
interactive "as-needed" basis. Thus, the information is created dynamically. 
The protection module 802 functions to prevent the sending of messages that 
may be construed as sexual harassment, racist, ethicist, or any similarly 
offensive and potentially illegal remarks. Module 803 may be utilized as an age 
detection module either by voice recognition, or the general flow of textual 
data may assist in determining the age of a user. The protection module 802 is 
connected to the topic analyzer 804 which determines if a specific message 
may be construed as offensive. Although this is not a major theme of our 
invention, we chose to mention it here. 

Figure 9 is a flow diagram showing the invention methodology. The 
process begins in function block 900 when a message is received through local 
means via a computer. The topic of a message is identified in function block 
901, and the time of a message is identified in function block 902. A 
determination is made in decision block 903 as to whether the topic has 
changed. If so, it is again necessary to identify the topic in function block 904. 
A determination is made in decision block 906 as to whether the topic is new. 
If the topic is new, then the system must create a new display window in 
function block 908. The message is then printed in the window in function 
block 909. If the different topic is not new, as determined in decision block 
906, then the system finds the window pertaining to the current topic in 
function block 907 and prints this message in that window. If the topic did not 
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change, as determined in decision block 903, then the system places the text 
from the message in the same window as displayed the previous message in 
function block 905. 

While the invention has been described in terms of a single preferred 
embodiment, those skilled in the art will recognize that the invention can be 
practiced with modification within the spirit and scope of the appended claims 
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